Remove python docstrings and Single-line comments #81
Conversation
- Updated to enhance comment removal logic. - Improved handling of Python docstrings and hash comments. - Refactored regex patterns for better accuracy. - Updated test cases in to reflect changes. - Ensured consistent use of double quotes in strings.
- Removed and statements from class.
WalkthroughThe changes introduced enhance the Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Files selected for processing (2)
- src/core/file/fileManipulator.ts (4 hunks)
- tests/core/file/fileManipulator.test.ts (23 hunks)
Additional context used
Biome
src/core/file/fileManipulator.ts
[error] 60-60: Use !== instead of !=.
!= is only allowed when comparing againstnull!= is only allowed when comparing against null
Using != may be unsafe if you are relying on type coercion
Unsafe fix: Use !==(lint/suspicious/noDoubleEquals)
[error] 163-163: Template literals are preferred over string concatenation.
Unsafe fix: Use a template literal.
(lint/style/useTemplate)
[error] 169-169: Template literals are preferred over string concatenation.
Unsafe fix: Use a template literal.
(lint/style/useTemplate)
[error] 177-177: Reassigning a function parameter is confusing.
The parameter is declared here:
Use a local variable instead.
(lint/style/noParameterAssign)
[error] 178-178: Reassigning a function parameter is confusing.
The parameter is declared here:
Use a local variable instead.
(lint/style/noParameterAssign)
Additional comments not posted (16)
tests/core/file/fileManipulator.test.ts (16)
Line range hint
153-173: Test case "Python comment, docstring removal" effectively validates comment and docstring removal.This test case ensures that both single-line comments and docstrings are properly removed from Python code, enhancing test coverage.
175-191: Test case "Python docstring removal mixing string declaration" correctly handles mixed string declarations.This test validates that string literals containing triple quotes are preserved while actual docstrings are removed.
193-213: Test case "Python comment f-string is not removed" ensures f-strings are preserved.This test confirms that f-strings and their contents remain intact during comment and docstring removal.
215-233: Test case "Python comment multi-line string literal is not removed" accurately preserves multi-line string literals.The test verifies that multi-line string literals are not mistaken for docstrings and are left unmodified.
235-247: Test case "Python nested quotes" correctly handles nested quotes within docstrings.This ensures that docstrings with nested single or double quotes are removed without affecting the integrity of string literals.
249-267: Test case "Python nested triple quotes with different types" effectively tests nested docstrings.The test checks the removal of docstrings containing nested triple quotes of different types, ensuring robustness.
269-281: Test case "Python inline comments" confirms that inline comments are properly removed.This test ensures that comments following code on the same line are correctly eliminated without affecting the code itself.
283-297: Test case "Python multi-line statement with string" preserves multi-line strings and removes comments.This verifies that multi-line statements using backslashes are not disrupted during comment removal.
299-315: Test case "Python docstring with triple quotes inside string literals" ensures string literals are preserved.The test confirms that string literals containing triple quotes are not mistaken for docstrings.
317-341: Test case "Python mixed comments and docstrings" accurately removes all comments and docstrings.This comprehensive test covers multiple scenarios of comments and docstrings interleaved with code, ensuring thorough removal.
343-361: Test case "Python f-strings with triple quotes" preserves f-strings during comment removal.The test ensures that multi-line f-strings with expressions are not altered.
363-373: Test case "Python escaped hash in string" correctly handles escaped characters.This verifies that hashes escaped within strings are not treated as comment indicators.
375-393: Test case "Python nested function with docstrings" properly removes docstrings from nested functions.The test confirms that docstrings in both outer and inner functions are removed without affecting the function definitions.
395-413: Test case "Python comment-like content in string" preserves strings containing comment-like content.This ensures that strings with hash symbols are not inadvertently altered during comment removal.
415-430: Test case "Python docstring with backslashes" correctly handles backslashes within docstrings.The test verifies that backslashes, even when escaping triple quotes, do not interfere with docstring removal.
432-442: Test case "Python mixed single and double quotes" preserves string literals with mixed quotes.This test ensures that strings containing triple quotes are not mistaken for docstrings unless they are actual docstrings.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- src/core/file/fileManipulator.ts (4 hunks)
Additional context used
Biome
src/core/file/fileManipulator.ts
[error] 177-177: expected
,but instead found:Remove :
(parse)
[error] 177-177: Expected a semicolon or an implicit semicolon after a statement, but found none
An explicit or implicit semicolon is expected here...
...Which is required to end this statement
(parse)
[error] 182-182: This code is unreachable
... because this statement will return from the function beforehand
(lint/correctness/noUnreachable)
Additional comments not posted (2)
src/core/file/fileManipulator.ts (2)
43-99: LGTM!The
removeDocStringsfunction looks good. It correctly handles both single and double quote docstrings, and the logic to accumulate lines in a buffer until a complete docstring is found is sound.
101-174: LGTM!The
removeHashCommentsfunction looks good. The binary search approach to determine if a hash index falls within a string literal is efficient, and the logic to process the content in a loop, identifying hash comments and appending the relevant portions to the result string is correct.
|
Thank you for this impressive work on improving our Python comment removal functionality. I really appreciate the effort you've put into this. I'll take some time to review the changes in detail. Your work looks promising, and I'm looking forward to diving into it. I've made a few commits to address some lint errors that came up. Please take a look at these changes when you have a moment. |
There was a problem hiding this comment.
Actionable comments posted: 2
Outside diff range and nitpick comments (6)
tests/core/file/fileManipulator.test.ts (6)
Line range hint
153-173: LGTM! Consider adding a more complex test case.The test case effectively covers basic scenarios for Python comment and docstring removal. The expected output correctly removes both single-line comments and docstrings.
Consider adding a more complex test case that includes nested functions with comments and docstrings to ensure the removal works correctly in more intricate code structures.
192-213: LGTM! Good test for f-string preservation.This test case effectively ensures that f-strings are preserved while comments are removed. Both single-line and multi-line f-strings are correctly handled.
Consider adding an inline comment within an f-string to test if it's correctly preserved, e.g.,
f'value: {x # comment}'.
234-247: LGTM! Good test for nested quotes in docstrings.This test case effectively handles the scenario of nested quotes within a docstring. The expected output correctly removes the entire docstring, including the nested quotes.
Consider adding a test case with a mix of single, double, and triple quotes to ensure robustness in handling various quote combinations.
268-281: LGTM! Good test for inline comment removal.This test case effectively verifies the removal of inline comments while preserving the actual code. The expected output correctly removes the comments without affecting the variable assignments.
Consider adding a test case with an inline comment containing a hash symbol within a string, e.g.,
x = "text # not a comment" # actual comment, to ensure proper handling of such scenarios.
316-341: LGTM! Comprehensive test for mixed comments and docstrings.This test case effectively covers a wide range of scenarios including single-line comments, docstrings, and inline comments within a single function. The expected output correctly removes all types of comments and docstrings while maintaining the code structure.
Consider adding a nested function within this test case to ensure that comment and docstring removal works correctly in nested structures as well.
431-443: LGTM! Excellent test for mixed quote types and docstring removal.This test case effectively verifies the correct handling of strings with mixed quote types and the removal of actual docstrings. The expected output accurately preserves the strings containing quote characters while removing the true docstring.
Consider adding a test case with nested quotes within a docstring, e.g.,
"""This "docstring" has 'nested' quotes""", to ensure proper handling of such complex scenarios.
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- src/core/file/fileManipulator.ts (1 hunks)
- tests/core/file/fileManipulator.test.ts (2 hunks)
Additional comments not posted (13)
src/core/file/fileManipulator.ts (2)
155-159: LGTM! Effective use of new methods.The updated
removeCommentsmethod correctly utilizes the newly implementedremoveDocStringsandremoveHashCommentsmethods. The order of operations is appropriate, removing docstrings before hash comments. The use ofrtrimLinesat the end ensures consistent formatting of the final output.
42-159: Overall, excellent implementation of Python comment removal functionality.The changes in this file effectively address the PR objectives by implementing robust methods for removing Python docstrings and single-line comments. The new
removeDocStringsandremoveHashCommentsmethods are well-designed, considering various edge cases and potential optimizations. The integration of these methods into the existingPythonManipulatorclass is seamless and maintains the overall structure of the codebase.Key strengths of the implementation:
- Efficient handling of both single and double quote docstrings.
- Preservation of comments within string literals.
- Use of binary search for optimizing string literal checks.
- Clear separation of concerns between docstring and hash comment removal.
The code is well-structured and maintainable, aligning with best practices for TypeScript development. The suggested optimizations for handling large files and improving string search efficiency could further enhance the performance of these methods.
tests/core/file/fileManipulator.test.ts (11)
174-191: LGTM! Good test for distinguishing strings and docstrings.This test case effectively checks the ability to distinguish between string declarations and docstrings. The expected output correctly preserves the string variable while removing the docstring.
214-233: LGTM! Proper handling of multi-line string literals.This test case effectively verifies that multi-line string literals are preserved and not mistaken for docstrings or comments. The expected output correctly maintains the entire string intact.
248-267: LGTM! Excellent test for complex nested quotes in docstrings.This test case effectively handles a complex scenario with nested triple quotes of different types within a function docstring. The expected output correctly removes the entire docstring, including all nested quotes, while preserving the function structure.
282-297: LGTM! Proper handling of multi-line statements with string concatenation.This test case effectively verifies that multi-line statements using string concatenation with the line continuation character () are preserved. The expected output correctly maintains the multi-line string intact while removing the comment after it.
298-315: LGTM! Excellent test for distinguishing docstrings from string literals with triple quotes.This test case effectively verifies the correct handling of docstrings and string literals containing triple quotes. The expected output accurately removes the actual docstring while preserving the string literals that contain triple quotes, demonstrating the ability to distinguish between these similar constructs.
342-361: LGTM! Good test for f-strings with triple quotes.This test case effectively verifies the preservation of multi-line f-strings using triple quotes while correctly removing comments. The expected output maintains the integrity of the f-string, including its internal calculation, and removes the comment following it.
362-373: LGTM! Proper handling of escaped hash in strings.This test case effectively distinguishes between escaped hash symbols in strings and actual comments. The expected output correctly preserves the string containing the escaped hash while removing the real comment, demonstrating the ability to handle potentially ambiguous situations.
374-393: LGTM! Excellent test for nested functions with docstrings.This test case effectively verifies the removal of docstrings in nested function structures. The expected output correctly removes both the outer and inner function docstrings while maintaining the overall function structure and logic.
394-413: LGTM! Thorough test for comment-like content in strings.This test case effectively verifies the preservation of strings containing comment-like content. It covers various scenarios including single-line and multi-line strings with hash symbols. The expected output correctly maintains all these strings intact, demonstrating the ability to distinguish between actual comments and comment-like content within strings.
414-430: LGTM! Good test for docstrings with backslashes.This test case effectively handles the scenario of docstrings containing backslashes. The expected output correctly removes the entire docstring, including the parts with backslashes, demonstrating robustness in handling escape characters within docstrings.
Line range hint
153-443: Excellent expansion of Python-specific test cases!The additions to the test suite significantly enhance the coverage for Python comment and docstring removal. The new test cases effectively cover a wide range of scenarios including:
- Basic comment and docstring removal
- Handling of f-strings
- Multi-line string literals
- Nested quotes in docstrings
- Inline comments
- Multi-line statements with string concatenation
- Docstrings vs string literals with triple quotes
- Mixed comments and docstrings
- Escaped hashes in strings
- Nested functions with docstrings
- Comment-like content in strings
- Docstrings with backslashes
- Mixed single and double quotes
These comprehensive tests will help ensure the robustness of the comment removal functionality for Python files.
|
HI, @thecurz ! Your attention to detail and the comprehensive test cases you've added are much appreciated. I'm going to go ahead and merge this PR. Thanks again for your valuable contribution! On a related note, I've noticed that our fileManipulator is growing quite large. We might consider splitting it into separate files in the future for better maintainability. |
Summary
Altered files
src/core/file/fileManipulator.ts:I added the methods removeDocStrings() and removeHashComments() to be used in removeComments() so that it's easier to implement an option to only delete certain types of comments later on.
tests/core/file/fileManipulator.test.ts:There are now 16 tests for Python comment removal.
Observations
Certain edge cases were not considered to avoid complexity.
For example in the code
only the first docstring is removed. But this is not standard practice in Python
Time Complexity
removeDocStringsFunction: O(m * n) where 'm' is the number of lines and 'n' is the average length of each lineremoveHashCommentsFunction: O((m * n) + m log m)Benchmark
For a better idea of the time complexity, I ran a quick test where the algorithm was applied to a repeated string of Python code containing several docstrings and comments.
For 3 million lines, the algorithm ran in around one second on my computer. This is more likely than not a decent time since LLMs won't take this many tokens anyway.
Next steps
I'll be happy to help with the implementation of any of these!
Summary by CodeRabbit
New Features
Bug Fixes
Tests